bead

a provenance tool

Krisztián Fekete

Miklós Koren

2025-09-10

The Editor Gives You One Week

You need to:

Address reviewer concerns about source data
Redo analysis with new data
Recreate Figure 1
Submit within one week

But the Submission Was Months Ago

Research submitted months ago
Team has been improving data cleaning since then
- Some team members left
Different statistical methods now
First question: How exactly was Figure 1 produced?

Research Results are Functions

\[\text{Figure 1} = \text{code}(\text{data})\]

Results depend on both algorithms and data
Code under version control (Git) ✓
Tagged commit at submission ✓
But what about the data?

Data is Also a Function

\[\text{data}_1 = \text{code}_2(\text{data}_2)\]

Data transformed by wrangling/cleaning steps
- countries dropped
- transformations applied
- feature engineering details
Chain of data provenance

A Real-World Data Pipeline

The Data Provenance Problem

Why It Is Complex:

Frequent changes: Code and data both evolve
Complex pipelines: Many steps, multiple datasets
Tool heterogeneity: Python, R, SQL, DuckDB all in one project
Team dynamics: People join, leave, change roles

Existing Solutions

Version Control (Git)

Great for code
Not suitable for large binary data

Data Version Control (DVC)

dvc.org

Similar spirit to bead, but delivery/versioning focused
More complex than needed for provenance tracking

Orchestration Tools

Apache Airflow (Python) - airflow.apache.org
dbt (SQL) - getdbt.com
KNIME (no-code) - knime.com
Too complex for heterogeneous teams

Enter bead

A command-line tool that captures your data’s story, step by step.

Much simpler than alternatives
Language agnostic
Supports heterogeneous teams

What bead Does NOT Do

Not a code runner

You run your own code
Python, R, Stata, SQL - doesn’t matter

Not a file delivery system

File system stores your files
You copy/move files yourself

Only requirement:

Works with flat files on file system
Files not too big (20GB works fine)

Core bead Concepts

The bead

Self-contained unit of computation
Contains code, reference to input data, results
Packaged as ZIP file
Remembers exact provenance

Simple Commands

bead new my-analysis
bead input add source-data
bead save

Demo Time

⚡ Inspiration

Data Journalism
Human concept of story
- Characters, setting, events, theme or message
- Variable scope
- Stories are composable
- Helps us to make sense of the world

📐 Design

ISBN / ISSN
- Identification of books and series with code
Task scope
Local first
Integrity

🎬 Demo

Conference session classification by title

Inputs:

list of sessions
session theme classification rules

Output:

list of sessions extended with theme

⚙️ Workspace and Box

/demo $

/demo $ : workspace - where we work

/demo $ : workspace - where we work /demo $

/demo $ : workspace - where we work /demo $ mkdir /demo/workspace

/demo $ : workspace - where we work /demo $ mkdir /demo/workspace /demo $

/demo/workspace

/demo $ : workspace - where we work /demo $ mkdir /demo/workspace /demo $ cd /demo/workspace

/demo/workspace

/demo $ : workspace - where we work /demo $ mkdir /demo/workspace /demo $ cd /demo/workspace /demo/workspace $

/demo/workspace

/demo $ : workspace - where we work /demo $ mkdir /demo/workspace /demo $ cd /demo/workspace /demo/workspace $ : box - storage for beads

/demo/workspace

/demo $ : workspace - where we work /demo $ mkdir /demo/workspace /demo $ cd /demo/workspace /demo/workspace $ : box - storage for beads /demo/workspace $

/demo/workspace

/demo $ : workspace - where we work /demo $ mkdir /demo/workspace /demo $ cd /demo/workspace /demo/workspace $ : box - storage for beads /demo/workspace $ bead box list

/demo/workspace

/demo/workspace

/demo/workspace

/demo/workspace

/demo/bead-box

/demo $ : workspace - where we work /demo $ mkdir /demo/workspace /demo $ cd /demo/workspace /demo/workspace $ : box - storage for beads /demo/workspace $ bead box list There are no defined boxes /demo/workspace $ mkdir /demo/bead-box /demo/workspace $ bead box add demo /demo/bead-box

/demo/workspace

/demo/bead-box

/demo/workspace

/demo/bead-box

/demo/workspace

/demo/bead-box

/demo/workspace

/demo/bead-box

📁 Output: Plain data

/demo/workspace $

/demo/workspace

/demo/bead-box

/demo/workspace $ bead new sessions

/demo/workspace

/demo/bead-box

/demo/workspace $ bead new sessions Created workspace "sessions" /demo/workspace $

/demo/workspace
└── sessions
    ├── input
    ├── output
    └── temp

/demo/bead-box

Workspace

name (sessions)
- input
- output
- temp

/demo/workspace $

/demo/workspace
└── sessions
    ├── input
    ├── output
    └── temp

/demo/bead-box

/demo/workspace $ cd sessions

/demo/workspace
└── sessions
    ├── input
    ├── output
    └── temp

/demo/bead-box

/demo/workspace $ cd sessions /demo/workspace/sessions $

/demo/workspace
└── sessions
    ├── input
    ├── output
    └── temp

/demo/bead-box

/demo/workspace
└── sessions
    ├── input
    ├── output
    └── temp

/demo/bead-box

/demo/workspace
└── sessions
    ├── input
    ├── output
    │   └── sessions.csv
    └── temp

/demo/bead-box

/demo/workspace
└── sessions
    ├── input
    ├── output
    │   └── sessions.csv
    └── temp

/demo/bead-box

/demo/workspace
└── sessions
    ├── input
    ├── output
    │   └── sessions.csv
    └── temp

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace
└── sessions
    ├── input
    ├── output
    │   └── sessions.csv
    └── temp

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace $ cd sessions /demo/workspace/sessions $ cat > output/sessions.csv <<EOF title,speaker Advances in Artificial Intelligence,Alice Blockchain Basics,Bob Sustainability in Business,Charlie Future of Quantum Computing,Diane Data Privacy and Security,Eric Machine Learning 101,Fiona AI Ethics and Impact,Grace Green Tech Innovations,Henry Distributed Ledger Technologies,Ian Effective Team Communication,Julia EOF /demo/workspace/sessions $ bead save Successfully stored bead at /demo/bead-box/sessions_20250910T150300641490+0000.zip. /demo/workspace/sessions $ bead discard Deleted workspace /demo/workspace/sessions /demo/workspace/sessions $ cd .. /demo/workspace $ : quick look into the created archive /demo/workspace $ unzip -v /demo/bead-box/sessions_*.zip | tail -9 ---- Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 332 Defl:N 236 29% 2025-08-31 22:29 4aa44c11 data/sessions.csv 245 Defl:N 171 30% 2025-09-10 15:03 80ce012f meta/bead 308 Defl:N 207 33% 2025-09-10 15:03 f4ddb35a meta/manifest -------- ------- --- ------- 885 614 31% 3 files /demo/workspace $ unzip -p /demo/bead-box/sessions_*.zip data/* | csvlook - | title | speaker | | ----------------------------------- | ------- | | Advances in Artificial Intelligence | Alice | | Blockchain Basics | Bob | | Sustainability in Business | Charlie | | Future of Quantum Computing | Diane | | Data Privacy and Security | Eric | | Machine Learning 101 | Fiona | | AI Ethics and Impact | Grace | | Green Tech Innovations | Henry | | Distributed Ledger Technologies | Ian | | Effective Team Communication | Julia | /demo/workspace $ bead new theme-aliases Created workspace "theme-aliases" /demo/workspace $

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    └── temp

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    └── temp

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    └── temp

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    └── temp

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
└── sessions_20250910T150300641490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

🛠 Calculation 1: CSV to DuckDB

/demo/workspace $

/demo/workspace

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-db

/demo/workspace

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-db Created workspace "sessions-db" /demo/workspace $

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-db Created workspace "sessions-db" /demo/workspace $ cd sessions-db

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-db Created workspace "sessions-db" /demo/workspace $ cd sessions-db /demo/workspace/sessions-db $

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-db Created workspace "sessions-db" /demo/workspace $ cd sessions-db /demo/workspace/sessions-db $ bead input add sessions

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   └── sessions
    │       └── sessions.csv
    ├── output
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   └── sessions
    │       └── sessions.csv
    ├── output
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    │   └── sessions.db
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    │   └── sessions.db
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    │   └── sessions.db
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    │   └── sessions.db
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-db Created workspace "sessions-db" /demo/workspace $ cd sessions-db /demo/workspace/sessions-db $ bead input add sessions Verifying archive /demo/bead-box/sessions_20250910T150300641490+0000.zip ... OK Loading new data to sessions ... Done /demo/workspace/sessions-db $ bead input add theme-aliases Verifying archive /demo/bead-box/theme-aliases_20250910T150520735490+0000.zip ... OK Loading new data to theme-aliases ... Done /demo/workspace/sessions-db $ cat > run.sh <<EOF duckdb output/sessions.db " -- Load sessions CREATE TABLE sessions AS SELECT * FROM read_csv_auto('input/sessions/sessions.csv'); -- Load aliases CREATE TABLE theme_aliases AS SELECT * FROM read_csv_auto('input/theme-aliases/theme_aliases.csv'); " EOF /demo/workspace/sessions-db $ bash run.sh /demo/workspace/sessions-db $ bead save Successfully stored bead at /demo/bead-box/sessions-db_20250910T150740610157+0000.zip. /demo/workspace/sessions-db $ bead discard Deleted workspace /demo/workspace/sessions-db /demo/workspace/sessions-db $ cd .. /demo/workspace $ unzip -v /demo/bead-box/sessions-db_*.zip | tail -9 Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 798720 Defl:N 2001 100% 2025-08-31 22:29 36fb105d data/sessions.db 249 Defl:N 132 47% 2025-08-31 22:29 1c671ac8 code/run.sh 937 Defl:N 478 49% 2025-09-10 15:07 bec88543 meta/bead 458 Defl:N 291 37% 2025-09-10 15:07 68cde410 meta/manifest -------- ------- --- ------- 800364 2902 100% 4 files /demo/workspace $

/demo/workspace

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

🔗 Calculation 2: Match Themes

/demo/workspace $

/demo/workspace

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-by-themes

/demo/workspace

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-by-themes Created workspace "sessions-by-themes" /demo/workspace $

/demo/workspace
└── sessions-by-themes
    ├── input
    ├── output
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-by-themes Created workspace "sessions-by-themes" /demo/workspace $ cd sessions-by-themes

/demo/workspace
└── sessions-by-themes
    ├── input
    ├── output
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-by-themes Created workspace "sessions-by-themes" /demo/workspace $ cd sessions-by-themes /demo/workspace/sessions-by-themes $

/demo/workspace
└── sessions-by-themes
    ├── input
    ├── output
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-by-themes Created workspace "sessions-by-themes" /demo/workspace $ cd sessions-by-themes /demo/workspace/sessions-by-themes $ bead input add sessions-db

/demo/workspace
└── sessions-by-themes
    ├── input
    ├── output
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    │   └── sessions_with_themes.csv
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    │   └── sessions_with_themes.csv
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    │   └── sessions_with_themes.csv
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    │   └── sessions_with_themes.csv
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ bead new sessions-by-themes Created workspace "sessions-by-themes" /demo/workspace $ cd sessions-by-themes /demo/workspace/sessions-by-themes $ bead input add sessions-db Verifying archive /demo/bead-box/sessions-db_20250910T150740610157+0000.zip ... OK Loading new data to sessions-db ... Done /demo/workspace/sessions-by-themes $ cat > run.sh <<EOF duckdb -readonly input/sessions-db/sessions.db " COPY ( SELECT s.title, s.speaker, LIST(DISTINCT a.canonical_theme) AS matched_themes FROM sessions s LEFT JOIN theme_aliases a ON LOWER(s.title) LIKE '%' || LOWER(a.alias) || '%' GROUP BY s.title, s.speaker ORDER BY s.speaker ) TO 'output/sessions_with_themes.csv' (HEADER, DELIMITER ','); " EOF /demo/workspace/sessions-by-themes $ bash run.sh /demo/workspace/sessions-by-themes $ bead save Successfully stored bead at /demo/bead-box/sessions-by-themes_20250910T151020352321+0000.zip. /demo/workspace/sessions-by-themes $ csvlook output/sessions_with_themes.csv | title | speaker | matched_themes | | ----------------------------------- | ------- | ----------------------------------------- | | Advances in Artificial Intelligence | Alice | [artificial intelligence] | | Blockchain Basics | Bob | [blockchain, artificial intelligence] | | Sustainability in Business | Charlie | [artificial intelligence, sustainability] | | Future of Quantum Computing | Diane | [quantum computing] | | Data Privacy and Security | Eric | [data privacy] | | Machine Learning 101 | Fiona | [artificial intelligence] | | AI Ethics and Impact | Grace | [artificial intelligence] | | Green Tech Innovations | Henry | [sustainability] | | Distributed Ledger Technologies | Ian | [blockchain] | | Effective Team Communication | Julia | [NULL] | /demo/workspace/sessions-by-themes $

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    │   └── sessions_with_themes.csv
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── sessions-by-themes
    ├── input
    │   └── sessions-db
    │       └── sessions.db
    ├── output
    │   └── sessions_with_themes.csv
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

✨ Versions: Inputs and Improvements

/demo/workspace $

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ : Create a new, supposedly improved version of

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ : Create a new, supposedly improved version of /demo/workspace $

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ : Create a new, supposedly improved version of /demo/workspace $ : the theme-aliases bead. For now it will be

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ : Create a new, supposedly improved version of /demo/workspace $ : the theme-aliases bead. For now it will be /demo/workspace $

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace $ : Create a new, supposedly improved version of /demo/workspace $ : the theme-aliases bead. For now it will be /demo/workspace $ : just an unchanged clone, with a later save date

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
└── theme-aliases_20250910T150520735490+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── theme-aliases
    ├── input
    ├── output
    │   └── theme_aliases.csv
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace $ : Create a new, supposedly improved version of /demo/workspace $ : the theme-aliases bead. For now it will be /demo/workspace $ : just an unchanged clone, with a later save date /demo/workspace $ bead edit theme-aliases --review Verifying archive /demo/bead-box/theme-aliases_20250910T150520735490+0000.zip ... OK Extracted source into /demo/workspace/theme-aliases /demo/workspace $ cd theme-aliases /demo/workspace/theme-aliases $ bead save Successfully stored bead at /demo/bead-box/theme-aliases_20250910T151240993101+0000.zip. /demo/workspace/theme-aliases $ bead discard Deleted workspace /demo/workspace/theme-aliases /demo/workspace/theme-aliases $ cd .. /demo/workspace $ : Use the "improved" input to "recalculate" a dependent /demo/workspace $ bead edit sessions-db Verifying archive /demo/bead-box/sessions-db_20250910T150740610157+0000.zip ... OK Extracted source into /demo/workspace/sessions-db Input data not loaded, update if needed and load manually /demo/workspace $ cd sessions-db /demo/workspace/sessions-db $ cat run.sh duckdb output/sessions.db " -- Load sessions CREATE TABLE sessions AS SELECT * FROM read_csv_auto('input/sessions/sessions.csv'); -- Load aliases CREATE TABLE theme_aliases AS SELECT * FROM read_csv_auto('input/theme-aliases/theme_aliases.csv'); " /demo/workspace/sessions-db $ bead input load Verifying archive /demo/bead-box/sessions_20250910T150300641490+0000.zip ... OK Loading new data to sessions ... Done Verifying archive /demo/bead-box/theme-aliases_20250910T150520735490+0000.zip ... OK Loading new data to theme-aliases ... Done /demo/workspace/sessions-db $

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    │   └── sessions.db
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    │   └── sessions.db
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    │   └── sessions.db
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace
└── sessions-db
    ├── input
    │   ├── sessions
    │   │   └── sessions.csv
    │   └── theme-aliases
    │       └── theme_aliases.csv
    ├── output
    │   └── sessions.db
    ├── run.sh
    └── temp

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

🖼️ Visual Overview

/demo/workspace $

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace $ rm /demo/bead-box/theme-aliases*

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
├── sessions_20250910T150300641490+0000.zip
├── theme-aliases_20250910T150520735490+0000.zip
└── theme-aliases_20250910T151240993101+0000.zip

/demo/workspace $ rm /demo/bead-box/theme-aliases* /demo/workspace $

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

/demo/workspace $ rm /demo/bead-box/theme-aliases* /demo/workspace $ bead box index rebuild /demo/bead-box

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

/demo/workspace

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

/demo/workspace $ rm /demo/bead-box/theme-aliases* /demo/workspace $ bead box index rebuild /demo/bead-box Rebuilding index for box directory: /demo/bead-box Index rebuild completed successfully /demo/workspace $ bead web color png session-themes.png Loaded bead 1 (demo / sessions @ 20250910T150300641490+0000) Loaded bead 2 (demo / sessions-db @ 20250910T150740610157+0000) Loaded bead 3 (demo / sessions-by-themes @ 20250910T151020352321+0000) Loaded bead 4 (demo / sessions-db @ 20250910T151520899203+0000) Loaded 4 beads Creating PNG: session-themes.png /demo/workspace $

/demo/workspace
└── session-themes.png

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

session-themes.png

⚙️ Internal Details

/demo/workspace $

/demo/workspace
└── session-themes.png

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

/demo/workspace $ unzip -p /demo/bead-box/sessions-by-themes_*.zip meta/bead

/demo/workspace
└── session-themes.png

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

/demo/workspace
└── session-themes.png

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

/demo/workspace
└── session-themes.png

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

/demo/workspace $ unzip -p /demo/bead-box/sessions-by-themes_*.zip meta/bead { "freeze_name": "sessions-by-themes", "freeze_time": "20250910T151020352321+0000", "inputs": { "sessions-db": { "content_id": "e17520bd10c89eef14449eee6a3d4668ea56b813d1ccb592495ddd1d0c75388a58c3d750ba284c35777c767eb9586a3dd4261511b020c0e14bf99ce353b9ff33", "freeze_time": "20250910T150740610157+0000", "kind": "f8796061879404a00023b0c5b722c16c-31665c88-499a-4fd7-b93b-ffd8801c6028" } }, "kind": "53cda5561e998bfdab098e582633b86b-de8a901c-1b1a-41d4-a8fe-d4c4dbced2ee", "meta_version": "aaa947a6-1f7a-11e6-ba3a-0021cc73492e" } /demo/workspace $ unzip -p /demo/bead-box/sessions-by-themes_*.zip meta/manifest { "code/run.sh": "f8e3890724fa40241a13908df2cfa87bf6ee61b7ab96bdaa017c9cc9829801a4728670a6a1b96afea7b5b7378216e71df87a490b48d9462f44eb32673c59ddd7", "data/sessions_with_themes.csv": "99fbbf6fb888b91d37bfeff52db4d1ff0617fdf286766adc60b0ac7edb9e3f2011aa6c38c4e7b01437ea365bb7733e283d0b1cba3f89840b8bb1bb32d86615b2", "meta/bead": "5dba428ea8e8ac8decc1d3877ff89da7b5eeb2085c13abb5d2da426261d5ce344f0cb789812d2d1fc03f8faaea114f049460273f9e2d6552779a5ab515b02110" } /demo/workspace $

/demo/workspace
└── session-themes.png

/demo/bead-box
├── sessions-by-themes_20250910T151020352321+0000.zip
├── sessions-db_20250910T150740610157+0000.zip
├── sessions-db_20250910T151520899203+0000.zip
└── sessions_20250910T150300641490+0000.zip

🧩 Similar ideas

kaggle notebook
- https://www.kaggle.com/code#:~:text=New,-Notebook
nix flake
- https://nixos.wiki/wiki/Flakes#Flake_schema
orderly2 (RSECON24 talk)
- https://youtu.be/lkPgihFQbrk

🐙 Source

https://github.com/e3krisztian/bead {{< qrcode https://github.com/e3krisztian/bead qrgithub 300 300 >}}

Case Study From Our Research Lab

Used since 2017 in our research groups: CEU MicroData, MACROMANAGERS.eu
- internally
- when sharing data with others
Saved about 600+ beads, two versions on average
Interquartile range of bead sizes: 10 to 500 MB, largest is 23 GB
Median time between saving new versions: 51 days

Practices We Adopted

Everything is a bead: raw data, intermediate data, analysis sample, research results
Never load data directly, from outside a bead
- Side product: always use relative paths
We don’t often recompute everything, but good to know we could

Lessons For Research Software Engineers

Minimal learning curve for researchers
No infrastructure requirements
Works with existing workflows
Complements version control
Enables true reproducibility

Key Takeaways

Data provenance is hard - especially with changing teams
Existing tools too complex - for heterogeneous research teams
bead keeps it simple - focuses on one thing well
Reproducibility becomes automatic - not an afterthought

Contact and Acknowledgements

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 313164). The views expressed are those of the authors and do not necessarily reflect those of the ERC or the European Commission.

References

World Development Indicators: data.worldbank.org/indicator
DVC (Data Version Control): dvc.org
Apache Airflow: airflow.apache.org
dbt: getdbt.com
KNIME: knime.com